Skip to content

feat(stt): add Azure AI Speech as an optional cloud speech-to-text provider#354

Open
amartya-dev wants to merge 1 commit into
calesthio:mainfrom
amartya-dev:feat/azure-speech-to-text
Open

feat(stt): add Azure AI Speech as an optional cloud speech-to-text provider#354
amartya-dev wants to merge 1 commit into
calesthio:mainfrom
amartya-dev:feat/azure-speech-to-text

Conversation

@amartya-dev

Copy link
Copy Markdown

Summary

Adds Azure AI Speech as an optional cloud speech-to-text provider. It is opt-in: when AZURE_SPEECH_KEY is configured, the agent may prefer azure_stt for cloud transcription, while the local faster-whisper transcriber remains the default offline path. Shared pipeline manifests are intentionally left unchanged, so no default provider selection is altered for existing users.

The tool uses Azure's Fast Transcription REST API, which accepts a local audio file directly (multipart upload) and returns a synchronous result with word-level timestamps and optional speaker diarization — no Azure Blob storage, SAS URLs, or async job polling. Its output schema mirrors transcriber exactly, so it is a drop-in for subtitle_gen and every other transcript consumer.

Related issue

No tracking issue — small additive provider integration.

Changes

  • tools/analysis/azure_stt.py — new azure_stt tool (capability=analysis, provider=azure, runtime=API). Calls the Fast Transcription endpoint, maps the response onto the existing transcript schema (segments + words[] + word_timestamps + language + duration_seconds). Follows the existing provider-tool conventions used by elevenlabs_tts: env-var get_status(), a _transcribe() helper wrapped for error handling, cost_usd/model on the ToolResult, and fallback="transcriber".
  • Auto-discovered by the registry — no registry or selector changes (the analysis capability has no selector).
  • tests/tools/test_azure_stt.py — contract, registry discovery, status behavior, the response→transcript mapping, execute() guardrails, and a mocked-network success path. No live API calls.
  • .agents/skills/azure-speech-to-text/ + .claude/skills/azure-speech-to-text/ — Layer-3 skill referenced by the tool's agent_skills.
  • docs/PROVIDERS.md — Azure AI Speech setup, API notes, and pricing; env-var summary entry.
  • .env.example, skills/INDEX.md, AGENT_GUIDE.md — document the optional cloud STT path alongside the default whisper transcriber.

Testing

  • make test373 passed, 8 skipped locally (18 new tests).
  • make test-contracts259 passed, 6 skipped.
  • make lint → clean (py_compile).
  • Verified end-to-end against a live Azure Speech resource: a short sample transcribed correctly with 19 word-level timestamps, en-US auto-detected, and cost_usd/model populated. Confirmed the output flows through subtitle_gen to produce valid SRT.

Checklist

  • The change is focused on a single logical concern.
  • I ran the relevant tests locally (make test-contracts / make test).
  • I updated docs/README where behavior or usage changed (docs/PROVIDERS.md, AGENT_GUIDE.md, skills/INDEX.md, .env.example).
  • No unrelated files (build artifacts, local config) are included in the diff.

Notes for reviewers

  • No default flip. azure_stt is additive and opt-in; pipelines and shipped config are untouched. Preference is expressed only when the user sets AZURE_SPEECH_KEY locally.
  • Fast Transcription, not Batch. Chosen because OpenMontage transcribes local files; Batch would require Blob-storage plumbing the repo doesn't have. Noted in the provider doc for files > ~2h.
  • Azure has no per-word confidence in this API, so each word carries its phrase confidence in probability (documented in the skill).

@amartya-dev amartya-dev requested a review from calesthio as a code owner July 10, 2026 17:58
…ovider

Add an Azure AI Speech transcription tool. It is opt-in: when
AZURE_SPEECH_KEY is configured the agent may prefer it for cloud STT,
while the local faster-whisper `transcriber` stays the default offline
path. Shared pipeline manifests are intentionally left unchanged, so no
default provider selection is altered for existing users.

- tools/analysis/azure_stt.py: new `azure_stt` tool (capability=analysis,
  provider=azure) calling the Fast Transcription REST API. The local file
  is uploaded via multipart and transcribed synchronously with word-level
  timestamps and optional diarization — no Blob storage or async polling.
  Output schema mirrors `transcriber` exactly, so it is a drop-in for
  `subtitle_gen` and other transcript consumers. Follows the existing
  provider-tool conventions (env-var status check, `_transcribe` helper,
  cost_usd/model on the result, fallback="transcriber").
- Auto-discovered by the registry; no registry or selector changes.
- tests/tools/test_azure_stt.py: contract, discovery, status, response
  mapping, execute guardrails, and a mocked-network success path (no live
  API calls).
- .agents/skills + .claude/skills: azure-speech-to-text Layer-3 skill.
- docs/PROVIDERS.md: Azure AI Speech setup, API notes, and pricing.
- .env.example, skills/INDEX.md, AGENT_GUIDE.md: document the optional
  cloud STT path alongside the default whisper transcriber.
@amartya-dev amartya-dev force-pushed the feat/azure-speech-to-text branch from 4e0f807 to a2a0d8c Compare July 10, 2026 18:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant